TST: Call tests just once with --dist=loadscope #28531

datapythonista · 2019-09-19T15:32:56Z

Another try to what was tried in #26949. Before this PR the tests are called twice, once in a core for the tests that affect shared results, and once in parallel for the rest.

This PR makes a single call, and tests in the same scope (class or module) are granted to run in the same core, so no shared data problems should happen.

The tests being very slow was possibly caused by the proxy env variables, and not --dist=loadscope. But please check how long tests took before merging.

datapythonista · 2019-09-19T16:10:44Z

If this finally works and can be merged, next step is to reimplement #26932. It failed before in the second run of the tests, but after this is merged, we'll have only one run, and it will work. That will make the clipboard tests run again in the CI.

datapythonista · 2019-09-20T09:21:39Z

There are some tests failing, I guess they use the same shared resources, but are not in the same scope, but the timings look ok.

Comparing times with #28542, those are the results:

Build	Original	dist=loadscope	diff
Linux p35_compat	25m	25m	0
Linux py36_32bit	25m	25m	0
Linux py36_locale_slow	24m	26m	+2
Linux py36_locale_slow_old_np	21m	26m	+5
Linux py37_locale	26m	25m	-1
Linux py37_np_dev	22m	23m	+1
Windows py36_np15	25m	26m	+1
Windows py37_np141	23m	22m	-1
macOS py35_macos	28m	22m	-6
travis 3.7	21m	23m	+2
travis 3.6, locale	23m	25m	+2
travis 3.6, coverage	28m	31m	+3
travis 3.6, slow	24m	22m	-2

I'll have a look at Linux py36_locale_slow_old_np to see if the increase on time is likely to be caused by the different parallelization of tests implemented here. And split some test files. Or if it's random noise or something else. But for the rest looks like there is no significant increase in time. So, I think it makes sense to move forward with this.

…iple

datapythonista · 2019-09-20T14:52:50Z

Looks like in test_sql.py there are files in different classes that use the same shared resources, so I changed the distribution of the tests to be per file, and not per scope.

I think we'll have to detect the file(s) that are taking longer in Linux py36_locale_slow_old_np and split them. Other than that, this looks good to me (there is an error in the allowed failures in Travis, but I guess it's unrelated).

See updated timings:

Build	Original	dist=loadscope	diff	dist=loadfile
Linux p35_compat	25m	25m	0	24m
Linux py36_32bit	25m	25m	0	26m
Linux py36_locale_slow	24m	26m	+2	28m
Linux py36_locale_slow_old_np	21m	26m	+5	30m
Linux py37_locale	26m	25m	-1	26m
Linux py37_np_dev	22m	23m	+1	23m
Windows py36_np15	25m	26m	+1	25m
Windows py37_np141	23m	22m	-1	23m
macOS py35_macos	28m	22m	-6	21m
travis 3.7	21m	23m	+2	26m
travis 3.6, locale	23m	25m	+2	26m
travis 3.6, coverage	28m	31m	+3	31m
travis 3.6, slow	24m	22m	-2	27m

WillAyd

Certainly simpler - this lgtm so I'd say merge if no other feedback and you have follow ups

jreback

i am weary of this let me look

…iple

datapythonista · 2019-10-01T14:32:32Z

Would be nice to get this merged somehow soon, so we can restore the clipboard tests. But this PR is a bit tricky and somehow risky, so let's review well, and re-run the CI couple of times before merging.

ci/run_tests.sh

jreback

I like the idea of this change, but I just don't want unexplained failures like we had before when we had resource contention. can you provide some insight on the question below.

ci/run_tests.sh

…iple

jreback

ok comment on deps; also should followup with a PR removing the marks themselves (single,multiple); let's not crowd this one. pls rebase as well. ping on green.

ci/run_tests.sh

jreback · 2019-10-03T17:26:02Z

ok one more rebase; ping on green.

datapythonista · 2019-10-04T02:19:42Z

Having this error in Travis:

QXcbClipboard::setMimeData: Cannot set X11 selection owner

................................................./home/travis/build/pandas-dev/pandas/pandas/compat/__init__.py:85: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.

  warnings.warn(msg)

/home/travis/build/pandas-dev/pandas/pandas/compat/__init__.py:85: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.

  warnings.warn(msg)

./home/travis/build/pandas-dev/pandas/pandas/compat/__init__.py:85: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.

  warnings.warn(msg)

/home/travis/build/pandas-dev/pandas/pandas/compat/__init__.py:85: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.

  warnings.warn(msg)

=================================== FAILURES ===================================

______________________ test_raw_roundtrip[\U0001f44d...] _______________________

[gw0] linux -- Python 3.6.9 /home/travis/miniconda3/envs/pandas-dev/bin/python

data = '👍...'

    @pytest.mark.single

    @pytest.mark.clipboard

    @pytest.mark.skipif(not _DEPS_INSTALLED, reason="clipboard primitives not installed")

    @pytest.mark.parametrize("data", ["\U0001f44d...", "Ωœ∑´...", "abcd..."])

    def test_raw_roundtrip(data):

        # PR #25040 wide unicode wasn't copied correctly on PY3 on windows

        clipboard_set(data)

>       assert data == clipboard_get()

E       AssertionError: assert '👍...' == ''

E         - 👍...

pandas/tests/io/test_clipboard.py:266: AssertionError

Not sure why this is failing here. Shouldn't be caused by the changes here. Has anyone seen this error before?

jreback · 2019-10-04T10:57:43Z

@datapythonista are we actually running the x clip tests currently? i thought was part of the issue

datapythonista · 2019-10-04T11:02:29Z

I thought we moved those tests to azure, where they were skipped. Not sure if the failing build was added after that. I'm in my phone, can't check in detail now.

…iple

jbrockmendel · 2019-10-28T19:46:17Z

@datapythonista still working on this?

datapythonista · 2019-10-29T14:49:24Z

Yes, still need to figure out what's the problem with the failing tests, but want to get this merged.

jreback · 2019-11-13T17:49:44Z

can you rebase when you have some time

…iple

jreback · 2019-11-16T21:52:43Z

it looks like the 3.6 coverage job is taking 50% longer than on master. can you confirm? (45 min)

maybe rebase and see if this repeats.

…iple

jreback · 2019-11-17T14:20:23Z

thanks @datapythonista

let's monitor the CI going forward for any odd failures

datapythonista · 2019-11-17T15:03:44Z

I created #29671 to speed up the builds.

jreback · 2019-11-17T15:07:32Z

great

in practice we are already splitting test files if they r too big (in loc)
but this is also a good metric

alimcmaster1 · 2019-11-17T18:08:03Z

Potentially related seeing this on travis 36 builds

[gw1] linux -- Python 3.6.6 /home/travis/miniconda3/envs/pandas-dev/bin/python
data = '👍...'
    @pytest.mark.single
    @pytest.mark.clipboard
    @pytest.mark.skipif(not _DEPS_INSTALLED, reason="clipboard primitives not installed")
    @pytest.mark.parametrize("data", ["\U0001f44d...", "Ωœ∑´...", "abcd..."])
    def test_raw_roundtrip(data):
        # PR #25040 wide unicode wasn't copied correctly on PY3 on windows
        clipboard_set(data)
>       assert data == clipboard_get()
E       AssertionError: assert '👍...' == ''
E         - 👍...
pandas/tests/io/test_clipboard.py:264: AssertionError

https://travis-ci.org/pandas-dev/pandas/jobs/613141276?utm_medium=notification&utm_source=github_status

https://travis-ci.org/pandas-dev/pandas/jobs/613153228?utm_medium=notification&utm_source=github_status

TST: Call tests just once with --dist=loadscope

9574dac

datapythonista added Testing pandas testing functions or related to the test suite CI Continuous Integration labels Sep 19, 2019

Remove code that was left by mistake

f772aa1

Marc Garcia added 2 commits September 20, 2019 14:36

Merge remote-tracking branch 'upstream/master' into remove_tests_mult…

c527e55

…iple

Replacing loadscope by loadfile

490e0b7

datapythonista mentioned this pull request Sep 28, 2019

CLN: Exception in io, plotting #28649

Closed

alimcmaster1 mentioned this pull request Sep 29, 2019

Use bash on windows #27195

Closed

1 task

WillAyd approved these changes Oct 1, 2019

View reviewed changes

jreback requested changes Oct 1, 2019

View reviewed changes

Merge remote-tracking branch 'upstream/master' into remove_tests_mult…

f93999a

…iple

jreback reviewed Oct 1, 2019

View reviewed changes

ci/run_tests.sh Show resolved Hide resolved

jreback requested changes Oct 1, 2019

View reviewed changes

ci/run_tests.sh Show resolved Hide resolved

datapythonista added 2 commits October 1, 2019 19:17

Merge remote-tracking branch 'upstream/master' into remove_tests_mult…

a7fdb62

…iple

Empty commit to rerun CI

432b352

jreback added this to the 1.0 milestone Oct 2, 2019

jreback requested changes Oct 2, 2019

View reviewed changes

ci/run_tests.sh Show resolved Hide resolved

ci/run_tests.sh Show resolved Hide resolved

jreback approved these changes Oct 3, 2019

View reviewed changes

datapythonista added 2 commits October 3, 2019 22:19

Merging from master

ff5148e

Pinning pytest-xdist

5c7575a

datapythonista added 2 commits October 12, 2019 15:55

Merge remote-tracking branch 'upstream/master' into remove_tests_mult…

1055fc7

…iple

Merge remote-tracking branch 'upstream/master' into remove_tests_mult…

5d49eba

…iple

datapythonista added 12 commits November 14, 2019 02:26

Merging from master

09618aa

Printing errors for xvfb-run

b848799

Not failing if xvfb is already running

8823919

Removing xvfb

699d7d9

Running xvfb only in travis

a8f15c3

Adding error information for xvfb

0c54fa0

Only run xvfb when DISPLAY is not defined

73934b6

Merge remote-tracking branch 'upstream/master' into remove_tests_mult…

9972e4d

…iple

Clarifying comment

a27609d

Removing xvfb in Travis config

2e79f38

Merge remote-tracking branch 'upstream/master' into remove_tests_mult…

bff544b

…iple

Merge remote-tracking branch 'upstream/master' into remove_tests_mult…

a4b1397

…iple

datapythonista added 2 commits November 16, 2019 23:51

Merge remote-tracking branch 'upstream/master' into remove_tests_mult…

5750713

…iple

Merge remote-tracking branch 'upstream/master' into remove_tests_mult…

bd100ac

…iple

jreback merged commit 940ea47 into pandas-dev:master Nov 17, 2019

datapythonista mentioned this pull request Nov 17, 2019

CI: Speed up coverage tests by splitting files #29671

Closed

alimcmaster1 mentioned this pull request Nov 17, 2019

CI: Clipboard Test Failures on Travis #29676

Closed

Reksbril pushed a commit to Reksbril/pandas that referenced this pull request Nov 18, 2019

TST: Call tests just once with --dist=loadscope (pandas-dev#28531)

6f23c45

jorisvandenbossche mentioned this pull request Dec 4, 2019

CI: the printing of skipped tests is broken #30040

Closed

proost pushed a commit to proost/pandas that referenced this pull request Dec 19, 2019

TST: Call tests just once with --dist=loadscope (pandas-dev#28531)

798e7db

proost pushed a commit to proost/pandas that referenced this pull request Dec 19, 2019

TST: Call tests just once with --dist=loadscope (pandas-dev#28531)

68394d8

jbrockmendel mentioned this pull request Dec 20, 2019

CI: troubleshoot codecov #30070

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TST: Call tests just once with --dist=loadscope #28531

TST: Call tests just once with --dist=loadscope #28531

datapythonista commented Sep 19, 2019

datapythonista commented Sep 19, 2019

datapythonista commented Sep 20, 2019 •

edited

Loading

datapythonista commented Sep 20, 2019

WillAyd left a comment

jreback left a comment

datapythonista commented Oct 1, 2019

jreback left a comment

jreback left a comment

jreback commented Oct 3, 2019

datapythonista commented Oct 4, 2019

jreback commented Oct 4, 2019

datapythonista commented Oct 4, 2019

jbrockmendel commented Oct 28, 2019

datapythonista commented Oct 29, 2019

jreback commented Nov 13, 2019

jreback commented Nov 16, 2019

jreback commented Nov 17, 2019

datapythonista commented Nov 17, 2019

jreback commented Nov 17, 2019

alimcmaster1 commented Nov 17, 2019

TST: Call tests just once with --dist=loadscope #28531

TST: Call tests just once with --dist=loadscope #28531

Conversation

datapythonista commented Sep 19, 2019

datapythonista commented Sep 19, 2019

datapythonista commented Sep 20, 2019 • edited Loading

datapythonista commented Sep 20, 2019

WillAyd left a comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

datapythonista commented Oct 1, 2019

jreback left a comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

jreback commented Oct 3, 2019

datapythonista commented Oct 4, 2019

jreback commented Oct 4, 2019

datapythonista commented Oct 4, 2019

jbrockmendel commented Oct 28, 2019

datapythonista commented Oct 29, 2019

jreback commented Nov 13, 2019

jreback commented Nov 16, 2019

jreback commented Nov 17, 2019

datapythonista commented Nov 17, 2019

jreback commented Nov 17, 2019

alimcmaster1 commented Nov 17, 2019

datapythonista commented Sep 20, 2019 •

edited

Loading